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ABSTRACT 


*■» 


This  is  a  report  on  the  criterion  development  phase  of  a  long  term 
research  program  concerned  with  computer  personnel  selection  and  evaluate 

Two  types  of  criterion  measures  are  involved  in  this  phase.  Both 
are  proficiency  tests,  one  designed  to  test  an  individual's  knowledge  of 
the  basic  principles  and  techniques  of  programming,  and  the  second,  to 
test  an  individual's  performance  in  depth  in  the  systems  analysis  and 
systems  design  areas.  The  development  of  the  first  type,  the  Basic 
Programming  Knowledge  rest  (BPKT) ,  is  described  in  this  report.  The 
second  type  of  measure  is  now  under  construction  and  will  be  described 
in  a  subsequent  report. 

The  BPKT  is  intended  to  stand  by  itself  as  a  criterion  of  program¬ 
ming  proficiency.  To  achieve  a  close  correspondence  of  test  content  t*' 
programming  job  requirements,  subject-matter  experts  participated  in 
the  construction  and  review  of  the  test  questions.  Test  questions  were 
selected  that  met  the  criteria  of  discrimination  and  appropriate  diffi¬ 
culty^  as  indicated  by  the  statistical  analysis  of  results  of  a  large 
preliminary  testing.  The  final  form  of  the  test  consists  of  100  multiple 
choice  quesxions  that  are  designed  to  be  free  of  references  to  specific 
computers  and  languages  now  in  use. 

Normative  scores  have  been  developed  for  Navy  computer  groups.  The 
relationships  of  the  BPKT  test  scores  to  a  number  of  vocational  and 
educational  variables  are  described.  Recommendations  are  made  tor  the 
use  of  the  BPKT  in  personnel  selection  and  evaluation  of  experienced 
prograasssrs  and  analysts,  and  as  a  criterion  measure  against  which 
aptitude  tests  may  be  validated. 
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COMPUTER  PERSONNEL  SELECTION  AND  CRITERION  DEVELOPMENT: 

III.  THE  BASIC  PROGRAMMING  KNOWLEDGE  TEST 

SECTION  I.  INTRODUCTION 

This  is  the  third  report  of  a  long-term  research  program  devoted 
to  computer  personnel  selection  and  evaluation.  The  program  is  divided 
into  three  phases:  1)  job  analysis  -  to  describe  what  programmers  and 
analysts  do;  2)  criterion  development  -  to  measure  the  effectiveness  of 
experienced  programmers  and  analysts;  and  3)  aptitude  test  development  - 
to  select  inexperienced  individuals  for  training. 

The  job  analysis  phase  was  described  in  two  previous  technical 
reports  (Rigney,  Berger,  Gershon,  1963;  Rigney,  Berger,  Gershon,  Wilson, 
1963).  This  report  is  concerned  with  the  criterion  development  phase, 
specifically  the  building  of  a  proficiency  test,  the  Basic  Programming 
Knowledge  Test  (BPKT),  as  one  type  of  criterion  measure  of  programing 
performance. 

There  are  various  kinds  of  criteria  In  use  for  evaluating  computer 
personnel,  but,  as  in  other  technical  fields,  most  of  these  criteria 
are  not  entirely  satisfactory.  Criteria  that  depend  on  subjective 
evaluation  by  supervisors,  on  salary  and  experience  levels,  on  job 
classification,  or  on  productivity  and  error  rate  measures,  sre  either 
inappropriate,  unreliable,  ci  lack  standardisation  from  company  to  com¬ 
pany  or  even  from  sect ioe  to  section  within  a  company.  It  is  proposed 
that  a  way  out  of  this  dilemma  is  to  use  a  proficiency  test  as  a 
criterion  measure.  This  report  describes  the  development  of  such  a 


test,  involving  a  comprehensive  job  analysis,  statistical  analysis  of 
test  scores,  and  review  by  the  intended  user  at  all  stages. 

The  8PKT  was  planned  to  meet  three  needs.  The  first  was  for  an 
instrument  to  use  for  selecting  experienced  personnel;  the  second  was 
for  a  method  to  assist  in  classifying,  evaluating,  and  upgrading  pro¬ 
grammers  and  analysts;  and  tne  third  was  for  ar  objective,  reliable 
research  instrument  to  be  used  in  validating  aptitude  tests  or  other 
predictors. 

1 

The  management  of  NAVCOSSACT  expressed  the  need  for  an  objective 
method  to  aid  in  the  selection  and  in  the  evaluation  and  classification 
of  applicants  for  programmer  and  analyst  positions.  The  computer  fiel 
has  drawn  to  it  a  vast  assortment  of  individuals  of  varying  education 
and  experience.  There  is  no  standard  computer  training  curriculum,  and 
many  of  the  people  doing  programming  or  analysis  learned  the  skills 
partially  or  completely  on  the  job.  In  addition,  people  vary  in  their 
ability  to  acquire  programming  or  analyst  skills,  and  in  their  ability 
to  transfer  what  skills  they  have  acquired  from  cne  situation  to  another. 
NAVCOSSACT  supervisors  needed  to  supplement  subjective  evaluation  of  an 
applicant  with  some  objective  arsesement  of  his  programing  proficiency, 
particularly  those  skills  most  useful  in  NAVCOSSACT  work.  The  BPKT  was 
constructed  to  measuro  those  skills. 

lThia  is  the  Naval  Command  Systems  Support  Activity.  NAVCOSSACT 
is  a  computer-centered  activity  that  supports  the  Chief  of  Naval  Operatic 
program  of  establishing  and  maintaining  an  Automatic  Data  Processing 
System  Network  for  Naval  cossnand  and  control  systems.  The  systems  analy* 
and  programing  efforts  in  the  various  ADP  projects  are  carried  out  by 
coordinated  teams  of  military,  civil  service,  and  contractor  personnel 
at  NAVCOSSACT. 


The  BFKT  measures  proficiency  primarily  at  the  lower  and  inter¬ 
mediate  programming  levels.  Plans  for  a  second  type  of  proficiency 
test,  specific  to  the  areas  of  systems  analysis  and  systems  design  are 
discussed  in  Section  V  of  this  report. 

The  research  use  of  the  BPKT  as  a  criterion  against  which  aptitude 
tests  and  other  predictors  may  be  validated  will  be  an  important  part  of 
the  overall  research  program.  A  battery  of  aptitude  tests  has  been 
systematically  administered  to  starting  classes  of  trainees  in  a  Navy 
programming  course.  The  tests  will  be  correlated  later  against  results 
with  the  BPKT  given  to  the  same  personnel  when  they  are  on  the  job. 
Aptitude  tests  were  also  administered  to  participants  in  the  norming 
of  the  BPKT  to  collect  concurrent  validity  information.  The  results 
of  this  latter  testing  are  discussed  in  Section  III. 


SECTION  II.  TEST  DESIGN  AND  DEVELOPMENT 


The  Basic  Programming  Knowledge  Test  was  designed  to  sample  a 
broad  range  of  programming  information,  techniques,  and  applications. 
Fundamental  to  the  design  purpose  was  the  representation  of  the  most 
commonly  exhibited  programming  behavior  and  knowledge.  The  construction 
of  each  test  question  started  with  a  task  statement  or  set  of  task 
statements  that  had  been  shown  to  be  descriptive  of  a  part  of  most 
o^ogrammir.g  jobs.  Subsequent  development  of  the  test  questions  involved 
*  series  of  checks  on  their  suitability  for  eliciting  and  measuring 
the  described  programming  behavior  or  knowledge.  Since  the  validity 
of  the  BPKT  depends  in  large  measure  on  sampling  the  subject  matter 
that  underlies  job  performance,  the  procedure  for  achieving  this  con¬ 
tent  validity  is  of  importance. 


Test  Item  Specifications 

Xht  content  of  the  BPKT  was  based  on  those  programming  activities 
and  tasks  which  a  large  group  of  programmers  said  they  spend  a  moderate 
or  great  amourt  of  time  doing.  It  was  assumed  that  if  a  large  proportion 
of  programmers  say  they  do  a  particular  ta9k,  then  the  knowledge  and 
skill  required  for  performing  the  task  is  basic  -o  the  programing 
f ie Id . 

With  the  ass > stance  of  experts  experienced  in  computer  work  an 
effort  was  made  to  develop  an  exhaustive  list  of  statements  describing 
the  tasks  which  computer  personnel  perform.  Each  statement  begins  with 
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a  verb,  such  as  "develop"  or  "code"  or  "design"  and  consists  ol  one  or 
two  brief  sentences .  After  several  revisions  and  tryouts,  a  final  Hat 
of  186  task  statements  was  developed  as  the  basic  tool  tor  job  dcscrip- 
tion  and  classification.  Following  are  examples  of  task  statements: 

Code  in  compiler  language. 

Make  corrections  to  own  or  other's  programs  by  reassembly 

or  recompilation  of  corrected  source  - language  code. 

Develop  program  production  schedules. 

Assign  people  with  appropriate  skills  and  talents  to  tasks  that 

require  these  skills. 

A  pool  of  the  most  often  selected  task  statements  was  established 
by  using  job  analysis  data.  The  procedure  for  making  up  the  pool 
consisted  of  finding  the  task  statements  which  were  rated  by  fifty  per 
cent  or  more  of  the  programmer  group  as  activities  they  spent  a  moderate 
or  large  amount  of  time  doing.  This  procedure  was  first  carried  cut  with 
the  50  programmers  from  sixteen  organizations  that  participated  in  the 
job  analysis  phase  cf  the  research.  The  same  procedure  was  then  applied 
to  a  second  group  made  up  cf  2i  programmers  from  NAVCOSSACT. 

There  v*a  a  considerable  overlap  of  the  tasks  selected  by  both 

groups,  indicating  the.  general  applicability  of  a  test  based  on  these 

tasks  to  programming  personnel  in  groups  outside  of  NAVCOSSACT.  Since 

one  of  the  primary  intended  uses  of  the  3PKT  was  for  screening  pro- 

* 

grainming  applicants  to  NAVCOSSACT  whose  experience  v  is  obtained  elsewhere, 

^The  entire  list  of  186  task  statements  is  given  in  Technical 
Report  No.  36,  and  the  results  of  the  job  analysis  are  presented  in 
Technics 1  Rep or E  No.  37. 


-5- 


Che  underlying  communality  of  activities  tested  would  make  the  test  fair 
as  well  as  effective  as  a  selection  ‘nstrument.  The  pool  consisted  of 
59  task  statements  that  were  common  to  programming  jobs  in  as  well  as 
outside  NAVCOSSACT. 

Categories  of  Programming  Tasks 

This  pool  of  task  statements  was  divided  into  f'x  general  categories 
These  categories  provided  convenient  groupings  of  tasks  for  test  question 
construction.  The  s<x  categories  of  tasks  are  described  below; J 

1.  Logic ,  Estimation  and  Analysis.  These  tasks  involve  analyzing 
a  problem  into  its  component  parts  and  converting  them  into  a  logical 
and  efficient  sequence  of  operations,  ind  selecting  the  most  appropriate 
method  when  considering  computing  coats,  programming  ease,  or  the  user's 
requirements . 

2.  Flow  Diagram sing.  These  tasks  deal  with  interpreting  and 
constructing  both  general  and  detailed  functional  flow  diagrams  and 
program  flow  diagrams. 

3.  Programming  Constraints.  These  tasks  are  concerned  with  the 
constraints  imposed  on  progressing  by  computer  characteristics  such 
as  storage  allocation,  computing  speed,  end  date  format. 

4.  Coding  Operations.  These  tasks  deal  with  numeric  machine  code, 
assembly,  end  higher ‘level  languages.  Knowledge  of  various  coding 
techniques  and  their  relative  merits  is  Involved. 

5 .  Program  Testing  and  Checkins.  The  tasks  in  this  category 
Involve  test  planning,  progressed  checks,  debugging  by  use  of 
diagnostic  techniques  such  as  dumps,  traces,  and  snapshots,  and  code 

•*The  list  of  task  statements  in  each  category  is  presented  in 
Appendix  A. 


corrections  by  selecting  the  appropriate  octal  and  symbolic  sequences 
of  code  to  correct  a  program  error. 

6.  Documentation.  These  tasks  involve  the  preparation  of  various 
types  of  documentation  for  different  types  of  audiences,  such  as  manage¬ 
ment,  users,  operators,  designers,  and  program  maintenance  personnel. 

Test  Item  Format  and  Generality 

The  BPKT  is  intended  primarily  as  a  test  of  breadth  of  knowledge 
of  the  commonly  used  principles  and  techniques  of  programming.  The 
multiple-choice  format  lends  itself  well  to  making  this  kind  of  assess¬ 
ment.  Performance  as  well  as  knowledge  is  measured  to  some  degree 
by  the  multiple-choice  questions  in  the  BPKT.  The  solutions  to  many 
of  the  problems  require  programming  performance  behavior  (for  example, 
tracing  through  a  flow  diagram). 

Perhaps  the  most  difficult  part  of  constructing  and  revising  the 
test  questions  was  to  keep  them  free  of  references  to  any  particular 
computer  or  programming  language  currently  in  use.  Since  it  is  doubtful 
that  questions  relating  to  programming  aver  can  be  divorced  completely 
from  computer  characteristics  snd  language  logic,  an  attempt  was  made 
to  let  only  those  computer  characteristics  that  are  general  to  all 
computers,  or  to  the  most  commonly  used  coasters,  influence  the  test 
questions.  The  sane  was  done  with  respect  to  type  of  programming 
language . 

Test  questions  were  tried  out  on  programming  personnel  from  many 
different  organisations  with  a  variety  of  computer  types  and  programming 
languages  represented.  During  these  try-outs,  the  examinees  were  asked 
t  >  comment  on  any  test  questions  they  felt  required  special  knowledge 
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connected  with  a  particular  computer  or  language.  These  comments, 
along  with  item-analysis  information  derived  from  each  try-out  group, 
were  used  to  eliminate  or  to  revise  the  biased  questions.  The  test 
questions  were  further  screened  for  biasing  influences  by  two  inde¬ 
pendent  panels  of  computer  experts,  and  by  a  preliminary  testing  of 
the  BPKT  with  a  relatively  large  number  of  programmers  and  analysts 
from  several  different  organizations. 

Certain  kinds  of  programming  tasks  could  not  be  tested  without 
specifying  characteristics  of  a  computer  and  language.  The  device  of 
a  "hypothetical  computer"  was  employed  to  allow  everyone  to  start  out 
on  the  same  footing  in  attempting  the  problems,  and  to  provide  a 
specific  computer-context  for  the  questions.  A  separate  section  of 
the  BPKT  was  designed  in  which  the  questions  were  organized  around 
sequences  of  hypothetical  mnemonic  code  for  a  hypothetical,  binary 
computer . 


l££lI$S«  Selection 

QfYtlMfftfit  2l  1M  Hat  1SMM  £2Sl 

Approximately  500  test  questions  were  con  ributed  by  about  30 
prograsKrs  end  analysts.  Most  of  the  contributors  were  from  NAVCOSSACT. 
These  questions  were  reviewed  and  edited  by  a  group  of  three  programming 
consultants.  The  250  questions  which  survived  the  review  procedure  were 
then  pre-tested  In  sett  of  approxlsuitely  50  questions,  with  small  groups 
oi  programmers.  The  groups,  varying  in  size  from  14  to  40,  included 
protirarmrers  attending  computer  conferences  or  working  in  computer 
facilities  on  the  West  Coast. 


Different  sets  of  questions  were  administered  to  these  groups  and 
the  results  were  analyzed  to  discover  those  questions  that  were  too 
easy  or  that  failed  to  discriminate  in  the  right  direction  between 
high  scoring  and  low  scoring  individuals.  Many  test  questions  were 
revised  on  the  basis  of  the  item-analysis  results  and  the  examinees' 
comments  and  were  pretested  further. 

The  results  of  the  pretesting  were  reviewed  by  a  committee  of  12 
programmers  and  analysts  at  NAVCOSSACT.  The  committee  recomnended  183 
of  the  items  as  likely  candidates  for  inclusion  in  the  final  test  form. 

The  final  form  was  planned  to  consist  of  100  items. 

Preliminary  Testing  and  Analysis 

To  reduce  the  183  item  pool  to  100  items,  another  preliminary 
testing  was  carried  out.  This  time  to  get  greater  stability  and  re¬ 
liability  of  the  results,  a  relatively  large  sample  of  computer  personnel 
was  tested  on  all  183  items.  There  were  120  participants  drawn  from 
eight  organisations.  Fifty-one  were  classified  as  programmers  and  69 
as  senior  prograsnmrs  end  analysts.  The  programming  experience  ranged 
from  4  months  to  9  veers  with  a  median  of  3%  yeara.  Business,  scientific, 
and  military  systems  programming  areas  were  each  represented  by  about 
one  third  of  the  group. 

The  results  of  this  last  preliminary  testing  ware  analysed  by 
obtaining  difficulty  and  discrisdnatioo  indices  for  each  item.  These 
indices  were  used  as  the  primary  criteria  for  retaining  or  eliminating 
test  questions.  A  test  question  was  eliminated  as  too  easy  if  more 
than  80  per  cent  of  the  group  answered  it  correctly,  and  as  too  difficult 
if  less  than  30  per  cent  answered  it  correctly. 
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Each  test  question  that  met  the  difficulty  criterion  by  being  in 
the  30  per  cent  to  80  per  cent  range  was  then  inspected  to  see  if  it 
.aet  V'P  d if crimination  cri:erion.  The  sample  of  120  peopl*  was  divided 
equti'y  into  upper  and  lower  groups  on  the  basis  of  tne  total  sc  es 
made.  Questions  were  selected  for  the  final  fora  of  the  test  when  the 
proportion  of  high  scorers  getting  the  question  right  was  significantly 
larger  than  the  proportion  of  low  scorers  getting  the  question  right. 

The  statistical  significance  of  these  differences  was  determined  by 
computing  phi  coefficients  (Guilford,  1965). 

Description  of  the  Basic  Programming  Knowledge  Test 

The  Basic  Programming  Knowledge  Test  consists  of  100  multiple- 
choice  questions,  divided  into  two  sections.  Section  I  contains  75 
questions  that  range  over  many  kinds  of  programming  activities.  The 
content  has  been  generalized,  so  that  knowledge  of  a  particular  com¬ 
puter  or  programing  language  is  not  required  to  answer  the  questions. 
Section  II  consists  of  25  questions  oriented  around  a  hypothetical 
computer  and  a  given  set  of  computer  instructions.  A  summary  of  the 
char^~-ter: sties  of  the  hypothetical  computer  and  a  list  of  the  computer 
instructions  are  provided  the  examinee  in  a  separate  booklet  that  may 
be  referred  to  in  working  out  the  solutions  to  the  problems.  Coding 
and  debugging  activities  are  emphasised  in  this  second  section.  The 
examinee  is  required  to  work  out  each  solution  completely  and  accurately 
to  arrive  at  the  correct  answer. 

The  time  allowed  for  the  75  items  in  Section  I  1’  75  minutes.  The 
time  limit  for  the  25  items  in  Section  11  is  60  minutes,  including  the 
time  taken  to  look  over  the  summary  of  computer  characteristics  and  the 
instruction  list.  Since  the  items  are  not  arranged  in  order  of 


difficulty,  and  some  examinees  tend  to  spend  more  time  than  they  should 
on  the  difficult  ones,  the  time  left  to  finish  the  section  is  announced 
at  given  intervals. 

The  total  administration  time  for  the  two  sections  is  usually  2^ 
hours,  including  2 hours  of  actual  testing  time,  10  minutes  for  dis¬ 
tributing  the  test  booklets  and  reading  of  the  instruction  page  on  the 
front  of  the  booklet,  and  a  five  minute  break  between  the  two  sections 
of  the  test.  The  two  sections  were  designed  co  be  independent  of  each 
other,  separately  scored,  and  separately  timed.  Either  section  may  be 
given  first,  and  separate  testing  sessions  may  be  scheduled  for  the 
two  sections,  such  as  the  morning  and  afternoon  of  the  same  day,  or 
two  mornings  on  succeeding  days.  Depending  on  the  use  of  the  test,  the 
group  to  be  tested  and  the  time  limitations,  one  section  or  the  other 
may  be  dropped. 

Internal -consistency  reliability  estimates^  for  the  two  sections 
of  the  BPKT  were  computed  from  the  test  results  with  the  two  norming 
samples  described  below.  Section  I  reliability  astlswtas  for  the 
NAVCOSSACT  sample  and  the  general  Navy  sample  were  .86  and  .90, 
respectively.  The  Section  XX  reliability  estimates  for  the  same 
samples  were  .85  and  .68.  The  latter  reliability  value  is  probably 
an  underestimation  due  to  the  item  difficulty  of  Section  XI  for  the 
general  Navy  sample. 

The  correlation  of  Section  I  and  Section  XX  was  .49  for  NAVCOSSACT 
and  .51  for  the  general  Navy  sample.  This  Indicates  that  25  per  cent 

^The  internal-consistency  reliability  estimates  were  based  on 
Kuder-Richardson  formula  20. 
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of  what  the  two  sections  measure  is  related  (the  correlation  coefficient 
squared) .  Fcr  the  most  part  the  two  sections  are  measuring  different 
programming  knowledge  and  skills. 


Norming 

Normlng  is  a  procedure  in  which  a  scale  is  developed  to  provide  a 
meaningful  basis  for  interpreting  test  scores.  It  makes  it  possible  to 
determine  the  relative  standing  of  an  individual  within  his  reference 
group.  Teqt  scores  collected  for  the  group  are  established  along 
various  points  of  a  scale  from  highest  to  lowest,  and  any  individual's 
score  then  can  be  cvspsvsd  directly  with  these  scores.  A  percentile 
scale  was  used  in  the  norming  procedures  for  the  BPKT.  Thus,  a  per¬ 
centile  of  81  would  mean  the  examinee  made  a  score  better  than  the 
scores  of  81  per  cent  of  his  group.  Since  an  individual's  percentile 
score  could  be  high  with  one  group  and  lower  with  another,  it  is 
Important  that  the  norms  for  the  appropriate  group  be  used. 

Norms  for  the  final  form  of  the  test  were  developed  on  two  groups. 
The  first  was  a  sample  of  95  civilian  programmers  and  systems  analysts 
at  NAV008SACT ,  end  tne  second  was  a  sample  of  366  civilian  programmers 
and  systems  analysts  drawn  from  aaven  Navy  bureaus  at  eight  geographical 
locations. 

A  brief  description  of  the  norming  conditions  and  the  samples  is 
given  below.  The  actual  normative  data,  including  the  centlle  ranks, 
means,  median,  standard  deviation  and  score  range  for  each  sample  are 
given  in  a  data  supplement  to  this  report.  The  data  supplement  will 
have  a  restricted  distribution. 
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Norming  of  £  NAVCOSSACT  Sample 


Arrangements  were  made  at  NAVCOSSACT  to  test  as  many  programmers 
and  analysts  as  possible  over  a  two-day  period.  Each  testing  session 
lasted  a  half  day  and  two  groups  of  10  to  30  people  were  tested  in  each 
session.  In  addition  to  the  BPKT,  several  aptitude  tests  were  adminis¬ 
tered  to  collect  concurrent  validity  data  for  these  tests.  The  complete 
administration  tim^  for  the  BPKT,  the  aptitude  tests  and  the  vocational 
and  educational  data  form  was  four  hours.  The  testing  conditions  were 
excellent,  and  on  the  whole  the  motivation  of  the  individuals  during 
the  test  appeared  to  be  g'od. 

Well  over  half  of  the  NAVCOSSACT  personnel  actively  engaged  in 
computer  programming  or  aualysis  were  able  to  participate  in  the  testing 
There  were  129  people  in  all,  34  military  and  95  civil  service  personnel 
Contractor  personnel  were  not  included  in  this  testing.  Only  the  civil¬ 
ian  personnel  were  included  in  the  norm  sasq>le. 

Norming  of  a  General  Navy  Sample 

Computer  groups  in  Navy  installations  outside  of  NAVG08SACT  con¬ 
stituted  the  other  major  non  samp la.  In  collaboration  with  the  Office 
of  Industrial  Relations  of  the  Navy,  the  MKT  was  administered  to  458 
programmers  and  analysts  from  several  Navy  Commands5,  and  fro*  the 
Marine  Corps.  The  examinees  vara  located  In  headquarters  activities 
in  Washington,  D.C.  and  In  field  activities  In  Philadelphia,  Norfolk, 
Oakland,  San  Diego,  Point  Mugu,  and  Port  Hueneme. 


5The  ,<avy  Bureau  designations  at  the  time  of  testing  were  Ships, 
Yards  and  Docks,  Supplies  and  Accounts,  Weapons,  Personnel,  and  Finance. 
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Not  all  of  the  people  wh ■>  participated  in  this  testing  were  included 


in  the  development  of  the  norms.  Eliminated  from  the  norms  sample  were 
'.he  scores  of  40  individuals  with  less  than  1  year  of  programming 
e  ierience  or  whose  jobs  were  almost  entirely  managerial,  and  52  military 
personnel.  The  last  named  group  was  not  sufficiently  represented  in  the 
testing,  since  the  arrangements  were  directed  primarily  toward  the  Navy 
civilian  employees. 

The  BPKT  scores  for  the  general  Navy  sample  have  been  related  to 
other  data  collected  from  each  individual  at  the  time  of  testing.  A 
discussion  of  the  relationships  with  measures  of  experience,  training  and 
education,  and  aptitude  tests  is  given  in  the  next  section. 
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SECTION  III.  RELATIONSHIP  OF  THE  BPKT  TO  EXPERIENCE, 

TRAINING,  AND  APTITUDES 

Data  arc  presented  in  this:  section  showing  the  relationships  between 

scores  on  the  test  and  the  experience,  training  and  aptitudes  of  persons 

6 

who  have  taken  the  test.  The  data  discussed  are  for  the  general  Navy 
sample.7  These  data  provide  indirect  evidence  of  the  content  validity 
of  the  test.  If  the  test  measures  breadth  of  programming  knowledge, 
then  examinees  with  broader  experience  should  score  higher  on  the  test 
than  those  with  narrower  experience.  If  the  test  measures  level  of  pi o- 
gramming  skill,  then  those  examinees  ’ho  have  experience  in  jobs  requiring 
a  high  level  of  programing  skill  should  score  higher  than  those  who 
were  in  jobs  requiring  a  low  level  of  programming  skill. 

Results  of  the  analysis  are  presented  under  the  following  headings; 
Type  of  Experience,  Level  of  Experience,  Aaxnint  of  Experience,  Type  of 
Training,  Education  and  Aptitudes.  For  ease  of  interpretation,  the 
results  are  given  in  the  tables  as  percentages  of  examinees  who  scored 
above  the  median  of  the  total  group.  The  significance  of  differences  was 
evaluated  by  (two-tailed)  chi-square  median  tests  (Guilford,  1965).  The 
results  for  Section  1  of  the  BFKT  are  preseatea  first. 


A  copy  of  the  Educational  and  Vocational  Data  form  lo  given  in 
Appendix  B. 

7Data  for  the  NAVCOSSACT  sample  are  presented  in  a  data  supplement. 
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T.  ri-  of  Experience 

Type  of  experience  was  noted  by  asking  the  examinees  to  list  the 
computers  they  used,  the  programming  languages  they  used,  their  area 
of  programming,  and  their  primary  job  activity. 

Computer  Used 

Performance  on  Section  I  of  the  BPKT  was  related  to  the  type  of 
computer  with  which  the  individual  worked,  binary  or  decimal  (character). 
The  percentage  of  examinees  with  experience  on  binary  computers  who 
scored  above  the  median  on  Section  I  was  signficantly  larger  than  the 
percentage  of  those  with  experience  on  decimal  computers  only  (Table  1, . 
Sixty- four  per  cent  of  the  155  examineej  whose  work  involved  binary  or 
both  binary  and  decimal  machines  scored  above  the  overall  median.  Only 
40  per  cent  of  the  211  examinees  who  worked  solely  with  decimal  machines 
scored  above  the  overall  median.  The  probability  that  a  difference 
this  Urge  would  occur  by  chance  is  less  than  1  in  1000  (p  c  .001) . 


Examinees  were  categorised  as  to  whether  they  used  problem-oriented 
languages,  symbolic  and  numeric  languages,  or  both.  Table  1  shows  the 
percentage  of  eech  group  that  scorad  above  the  median. 

Thara  wart  significant  dlffarances  among  the  groups.  Examinees 
who  Indicated  they  used  only  a  problem-oriented  language  or  no  programmin 
language  were  least  likely  to  score  above  the  median.  Those  who  used 
symbolic  or  numeric  languages  were  about  equally  likely  to  score  high  or 
low.  Those  who  used  both  a  problem-oriented  language  and  a  symbolic  or 
numeric  la  "t*  were  mure  11‘ely  to  score  high. 
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Table  1 


BPKT  Section  1 

Relationships  with  Type  of  Experience 


Varic'bio 

N 

%  Above 
Median 

Chi 

Square 

P 

Computer  Used 

19.74 

<1.001 

Binary  or  Binary  &  Decimal 

155 

64 

Decimal  (character)  only 

211 

40 

Programming  Language  Used 

20.93 

<.001 

No  Language  Indicated 

32 

20 

Problem  Oriented  Language  (POL) 

51 

33 

Symbolic  or  Numeric 

144 

51 

POL  &  Symbolic  or  Numeric 

139 

61 

Area  of  Programing 

12.32 

<.01 

Data  Manipulation 

301 

47 

Utility  Programming 

35 

57 

Math  or  Scientific 

25 

84 

Primary  Job  Activity 

14.81 

<.001 

Systems  Analysis  k  Design 

93 

40 

Lead -Programming  & 

Supervising  .’rograa  Production 

51 

73 

Detailed  Program  Design  & 

Coding 

159 

47 

Area  of  Programming 

An  examinee's  area  of  programming  was  sign  f leant ly  related  to  his 
score  on  Section  1  of  the  BPKT.  Those  who  worked  in  mathematical  or 
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scientific  programing  were  much  more  likely  to  score  high.  Those  who 
worked  in  utility  programming  were  somewhat  more  likely  to  score  high, 
and  those  who  worked  in  data  manipulation  were  about  equally  likely  to 
score  high  or  low. 

Primary  Job  Activity 

The  examinees  were  asked  to  indicate  the  activity  that  described 
the  largest  part  of  their  job.  The  three  most  often-indicated  activities 
are  3hown  in  Table  1.  Examinees  whose  major  duties  were  systems  analysi- 
and  design  were  more  likely  to  score  low;  those  whose  major  duties  were 
lead  programming  or  supervising  program  production  activities  were  more 
likely  to  score  high;  and  those  whose  major  activities  were  detailed 
program  design  and  coding  were  about  equally  likely  to  score  high  or 
low. 

Some  of  those  whose  primary  activities  were  systems  analysis  and 
design  either  had  little  programing  experience  or  their  programming 
experience  was  r.cc  recent.  On  the  other  hand,  individuals  whose  primary 
activities  were  lead  programing  and  supervising  the  production  of  pro¬ 
gram  were  ifkaiy  to  have  bean  more  directly  Involved  in  programming 
activities  inquiring  a  high  level  of  programing  proficiency. 

Level  of  Experience 

Civil  Service  grade  and  job  titles  were  used  as  Indices  of  level 
of  experience,  It  ms  found  that  higher  civil  service  grade  levels 
were  somewhat  associatad  with  graater  programing  proficiency,  but 
when  job  titles  were  taken  into  account,  the  relation  between  grade 
level  and  programming  proficiency  Uid  not  always  hold.  The  general 
Navy  sample  was  a  heterogeneous  group  which  included  both  progranraers 
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and  analysts.  The  programmer  group  was  divided  into  programmers  and 
senior  programmers,  and  the  analyst  group  was  divided  into  those  who 
had  previous  programming  experience  and  those  who  did  not.  Many  of 
these  systems  analysts  were  employed  at  high  grade  levels  because  of 
their  superior  knowledge  of  programming1  many  others  were  at  high  grade 
levels  because  of  their  knowledge  of  a  particular  subject-matter  field 
rather  than  proficiency  in  programing. 

Civil  Service  Grade 

The  examinees  were  categorized  by  Civil  Service  grade  (GS  7  to  9, 
GS  11,  and  GS  12  and  above).  Those  examinees  in  the  higher  grades  were 
more  likely  to  score  high  on  the  test  than  those  in  the  lower  grades 
(p  <  .05)  as  shown  In  Table  2. 

Job  Title 

Examinees  were  also  categorized  according  to  the  job  titles  they 
listed.  Table  2  Indicates  that  prograanmrs  in  the  lower  grades  were 
about  equally  likely  to  be  high  or  lew  performers  on  Section  I  of  the 
BPXT.  Senior  programmers  (GS  114)  showed  a  slight  tsndcncy  to  score 
high.  Systesis  analysts  (GS  11+)  who  had  previously  held  the  title  of 
progr abater  were  slgnficantly  more  llkaly  to  acoro  high.  However, 
systems  analysts  (GS  114)  who  had  never  been  progtesssers  were  signifi¬ 
cantly  more  likely  to  score  low,® 


g 

Ten  systems  Analysts  were  at  the  GS  7-9  level  and  were  not  included 
in  the  job  title  analysis  because  of  the  small  size  of  the  group. 
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Table  2 


BPKT  Section  I 

Relationships  with  Level  of  Expedience 


Variable 

N 

%  Above 
Median 

Chi 

Square 

P 

Civil  Service  Grade 

6.14 

<•05 

GS  7  -  9 

84 

42 

GS  11 

166 

49 

GS  12  -  14 

116 

58 

Job  Title 

17.66 

<  .001 

Programmer  (GS  7-9) 

76 

47 

Senior  Programmer  (GS  11+) 

174 

56 

Systems  Analyst  (GS  11+, 
Previous  Programing  Title) 

52 

65 

Systems  Analyst  (GS  11+, 

No  Previous  Progreming  Title) 

54 

tmmmmmmm  -m 

28 

Anount  $1  Ixparlence 

Anount  of  experience  was  found  by  asking  each  axaminae  to  list  ell 
tba  programing  or  analyst  job  titlaa  ha  had  hold  (in  his  present 
organlaatlon  and  other  organisations)  and  the  number  of  months  he  had 
worked  with  each  job  title. 

of  fixperleacf 

Length  of  experience  was  determined  by  the  total  number  of  months 
the  individual  had  spent  in  computer-related  jobs.  There  was  a  slight 
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tendency  for  individuals  with  only  one  or  two  years  experience  to  score 
below  the  median  on  the  test. 

The  relationship  between  length  of  experience  and  performance  on 
Section  I  of  the  BPKT  is  not  a  linear  one.  The  complete  frequency  dis¬ 
tribution  cf  test  scores  plotted  against  the  distribution  on  experience 
by  years  shows  a  tendency  for  the  average  score  to  increase  from  the 
first  year  to  the  third  year  of  experience.  Beyond  the  third  year, 
the  average  score  for  each  year  tends  to  remain  about  the  same,  slightly 
above  average. 

Perhaps  the  reason  that  the  relationship  between  length  of  experience 
and  test  performance  was  not  more  pronounced  is  that  length  of  experience 
is  not  a  direct  index  of  level  of  skill  or  breadth  of  knowledge.  With 
increased  experience  and  seniority,  many  Individuals  move  away  from 
direct  concern  with  programming  to  othtr  typaa  of  activities.  Others 
work  in  prograaming  jobs  which  make  few  demands  for  learning  new  skills 
and  techniques.  Individuals  in  such  Jobs  are  in  the  situation  of  the 
five-year  man  who  has  had  one  year  of  experience  five  times. 

Number  of  Job  Titles 

Table  3  shows  a  highly  significant  relationship  between  performance 
on  Section  I  of  the  BPKT  and  the  nus*er  of  Job  titles  the  individual 
has  held.  The  greater  the  number  of  job  titles  held  the  greater  the 
likelihood  of  a  high  score.  It  should  be  noted  that  having  held  a 
large  number  of  job  titles  did  not  necessarily  represent  job-hopping. 


Table  3 


BPKT  Section  I 

Relationships  with  Amount  of  Experience 


Variable 

N 

7.  Above 
Median 

Chi 

Square 

P 

Computer-related  Experience 

4.63 

<.05 

1-2  years 

93 

41 

3  or  more  years 

273 

53 

Number  of  Computer-related 

Job  Titles 

27.77 

<.001 

1  Job 

128 

39 

2  Jobs 

125 

48 

3  Jobs 

64 

55 

4-6  Jobs 

44 

84 

In  many  instances  it  meant  that  the  individual  had  moved  up  through 
several  positions  in  a  single  organisation. 

Type  &£  Training 

The  examinees  were  asked  to  Indicate  whether  they  had  received 
training  in  General  Programing  Procedures,  Systems  Analysis,  Numerical 
Analysis,  Computer  Operation,  Machine  Languages  or  Machine-independent 
Languages.  Performance  on  Section  I  was  unrelated  to  training  in 
General  Programming  Procedures,  Systems  Analysis,  Numerical  Analysis, 
and  Computer  Operation.  However,  those  who  said  they  had  received 
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Table  4 


BPKT  Section  1 

Relationships  with  Type  of  Training 


Variable 

N 

7.  Above 
Median 

Chi 

Square 

P 

General  Programming  Procedures 

.68 

9 

n.s . 

Trained 

341 

51 

Untrained 

25 

44 

Systems  Analysis 

1.70 

n.s. 

Trained 

232 

53 

Untrained 

134 

46 

Numerical  Analysis 

3.72 

n.s. 

Trained 

64 

61 

Untrained 

302 

48 

Computer  Operation 

1.59 

n.s. 

Trained 

197 

53 

Untrained 

169 

47 

Machine  LansuAses 

34.16 

4  .001 

Trained 

267 

58 

Untrained 

79 

22 

Machine- independent  Languages 

22.06 

4.001 

Trained 

118 

68 

Untrained 

248 

42 

9 

n.s.  means  that  the  difference  is  not  significant  (p  >  .05). 
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training  in  Machine  Languages  or  in  Machine- independent  Languages  were 
significantly  more  likeN  to  score  high  than  were  those  who  did  not. 

These  results  are  summarized  in  Table  4. 

Education 

The  relationships  of  performance  on  the  BPKT,  Section  I  to  education 
variables  are  shown  in  Table  5.  It  is  evident  that  the  higher  the  level 


Table  5 

BPKT  Section  I 

Relationships  with  Education 


Variable 

N 

%  Above 
Median 

Chi 

Square 

P 

Level  of  Education 

49.45 

4.001 

High  School  or  less 

154 

29 

1-3  years  of  college 

97 

59 

Bachelor's  Degree 

86 

70 

Post-Graduate  Work 

29 

76 

College  Ha  lor 

6.54 

4.05 

Accounting  end  Business  Admin. 

62 

39 

Social  Science,  Humanities  &  Educ. 

50 

50 

Math,  Engr.,  Phys.  k  Biol.  Science 

58 

62 

of  education  or  the  more  science-oriented  the  college  major,  the  more 
probable  it  is  that  an  individual  will  score  above  the  median. 
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Aptitudes 


Several  groups  of  examinees  took  selected  aptitude  tests  in  addition 
to  the  BPKT.  The  test  names  are  listed  below  along  with  a  brief  descrip¬ 
tion  nl  tlu.  test  content  and  the  aptitude  measured  by  the  test. 


TEST  NAMES  AND  CONTENT 


APTITUDES 


'  •„  order  i ntt  -  List  steps  in  appropriate 

.  ,.i  »  omplcte  a  given  project. 

jymbo 1  croup  Ink  -  Rearrange  scrambled  symbols 
ii>  .1  '.pi  »  I  I  !«<t  order  as  efficiently  as  possible. 

Llliii:  Reason  ink  -  Discover  rules  for  marking 

•  i  !<•  in  patterns. 

I  tv  -  Matrix  -  Apply  rules  discovered  from 
i  hatigt  in  rows  and  columns  of  a  3  x  3  matrix 
o!  it  ties  to  select  a  figure  for  a  specified 
i  el  1 . 

Wo t_d  ii  instormatlon  -  Indicate  new  divisions 
b  .ween  letters  in  a  series  or  words  to  make 

*  new  scries  of  words. 


Logical  Ordering 

Symbol  Manipulation 

Discovering  Symbolic 
Patterns 

Figural  Relations 


Symbolic  Redefini 
tion 


Vocabulary  -  Select  the  word  that  la  most  Verbal  Comprehension 

similar  in  meaning  to  a  given  word. 


Four  of  the  aptitude  teats  showed  significant  relationships  to  per¬ 
formance  on  the  BPKT.  That  is,  individuals  woo  scored  in  the  upper 
half  on  these  aptitude  testa  were  more  likely  to  score  in  the  upper  half 
on  the  BPKT.  As  sunmarlsad  in  Tabls  6,  the  tests  which  showed  significant 
relationships  were  Temporal  Ordering,  Symbol  Grouping,  Circle  Reasoning 
and  Figure  Matrix.  These  results  imply  that  performance  on  Section  I 
of  the  BPKT  is  related  to  an  individual's  ability  to  reason  logically 
and  to  manipulate  abstract  symbols. 

The  vocabulary  test  was  administered  to  determine  whether  under¬ 
standing  directions  or  word  meanings  entered  into  performance  on  the  BPKT. 
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The  lack  of  a  significant  relationship  indicates  that  differences  in 
verbal  comprehension  ability  are  net  an  important  factor  in  BPKT  scores. 

Table  6 

BPKT  Section  I 

Relationships  with  Aptitude  Tests 


Variable 

N 

%  Above 
Medl an 

Chi 

Square 

P 

Temporal  Order ins 

10.31 

<.01 

Lower  Half 

27 

20 

Upper  Half 

29 

72 

Symbol  Grouping 

16.14 

<.001 

Lower  Half 

28 

25 

Upper  Half 

28 

79 

Circle  Reasoning 

4.32 

<.05 

Lower  Half 

45 

38 

Upper  Half 

48 

58 

Figure  Matrix 

14.16 

<.001 

Lower  Half 

43 

28 

Upper  Half 

49 

67 

sag 

0.18 

n.s. 

Lower  Half 

50 

50 

Upper  Half 

44 

48 

Vocabulary 

2.69 

n.s. 

Lower  Half 

76 

42 

Upper  Half 

73 

55 
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BPKT  Section  I_  Summary 

If  one  were  to  make  a  summary  stereotype  of  the  characteristics 
of  persona  who  are  likely  to  3core  above  average  on  Section  I  of  the 
BPKT,  one  would  say  that  such  persons  are  more  likely  to: 

1.  work  with  a  binary  computer 

2.  use  both  a  POL  and  a  symbolic  or  numeric  language 

3.  do  mathematical  or  scientific  programming 

4.  do  lead  programming  or  program  production  supervision 

5.  be  a  GS  12  or  above 

6.  be  a  senior  programmer  or  a  senior  systems  analyst  who  has 
formerly  been  a  programmer 

7.  have  had  at  least  3  years  experience 

8.  have  held  3  or  more  job  titles 

9.  have  had  training  in  machine  languages  and  machine 'independent 
languages 

10.  be  a  college  graduate 

11.  have  majored  in  mathematics,  engineering,  or  science 

12.  score  high  or  tests  of  logical  reasoning  and  symbol  manipulation 
abilities 

The  above  stereotype  should  be  interpreted  with  caution.  Like  most 
summaries,  it  does  not  do  justice  to  individual  cases,  since  it  Is  based 
on  differences  between  groups. 

Relationships  to  Section  II  of  the  BPKT 

Section  II  requires  interpreting  and  debugging  sequences  of  mnemonic 
code  for  a  hypothetical  computer.  The  relationships  of  experience, 
training  and  aptitudes  to  performance  on  this  section  of  the  test  are 
the  same  as  those  for  Section  I  with  the  following  exceptions: 


1.  GS  level  Is  not:  related  to  scores  on  Section  II. 

?..  Length  of  experience  is  not  related. 

3.  Systems  analysts  with  previous  programming  experience  are  ecuallv 
likely  to  score  high  or  low  on  Section  II.  They  were  more  likely 
to  score  high  on  Section  I. 

A.  Training  in  Computer  Operation  is  re  .ted  to  scores  on  Section 
II. 

5.  Scores  on  the  Vocabulary  test  are  related  to  scores  on  Section 
II.  The  Vocabulary  test  measures  verbal  comprehension  ability 
which  is  probably  important  in  reading  and  understanding  the 
instructions  for  the  hypothetical  computer. 
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SECTION  IV.  RECOMMENDATIONS  FOR  USE  OF  THE  BPKT 


It  was  the  authors'  intention  to  provide  a  criterion  of  programmer 
proficiency.  The  fulf Ills' .at  of  this  intent  rests  on  the  assumption  that 
because  of  the  elaborate  care  taken  in  the  translation  of  programming 
job  requirements  into  test  form,  the  test  stands  by  itself  as  a  criterion 
of  programming  proficiency.  Tie  acceptance  of  the  BPKT  as  a  criterion 
will  make  it  a  powerful  instrument  for  the  purposes  of  selection, 
evaluation,  and  placement  of  personnel.  Recommendations  with  respect  to 
the  uses  of  the  BPKT  are  given  below. 

Selection  of  Experienced  Personnel 

The  BPKT  is  recommended  for  an  Important  role  in  the  relection  of 
applicants  for  computer  programmer  positions.  The  test  will  be  particu¬ 
larly  useful  as  a  screening  device  when  there  are  a  large  number  of 
qualified  applicants  for  a  position.  Tha  number  of  people  to  be  Intel  - 
viewed  could  be  reduced  to  those  Individuals  who  are  above  a  given 
standard  of  technical  proficiency  as  determined  by  a  cut-off  score. 

For  the  selection  of  applicants  for  a  highar  level  position,  such 
as  systems  analyst,  the  BPKT  could  be  used  in  combination  with  tasts 
specially  designed  for  that  job  area.  Figure  1  presents  a  plan  for  using 
proficiency  tests,  singly  or  in  combination,  for  selacting  programmers 
and  systems  analysts.  The  selection  of  applicants  for  a  programming 
position  would  involve  the  use  of  the  -PKT  only.  Applicants  for  a  system 
analysis  position  can  be  differentiated  into  two  job  categories:  those 
whose  job  may  require  Involvement  in  the  programming  portion  of  a  project, 
and  those  whose  job  is  primarily  concerned  with  the  subject-matter 

applications  of  a  project. 
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Figure  1  outlines  a  sequential  decision  procedure.  The  BPKT  would 
be  the  first  stage  test  for  those  cases  in  which  programing  experience 
is  a  prerequisite.  Candidates  for  system  analysis  positions  who  scored 
higher  than  the  cut-off  score  would  be  given  special-area  tests  (SAT)  to 
obtain  additional  information  to  assist  in  tl.  •  placement  decision.  The 
phraso  "conditionally  accept"  in  the  figure  mean?  that  other  information, 
in  addition  to  test  scores,  may  be  available  to  management.  There  are 
characteristics  of  applicants  that  cannot  be  revealed  by  tests;  for 
example,  physical  appearance  and  skill  in  social  interaction.  To  the 
extent  that  management  emphasizes  the  importance  of  these  characteristics 
this  supplementary  information  will  be  important  in  determining  the  fina: 
decision  about  the  applicant.  These  recommendations  for  the  sequential 
application  of  criterion  measures  are  somewhat  analogous  to  sequential 
decision  strategies  discussed  in  Cronbach  and  Gleser  (1965). 

Evaluation  of  Personnel  On-the-Job 

The  BPKT  can  be  used  to  evaluate  computer  personnel  presently  em¬ 
ployed  who  are  being  considered  for  reassignment,  advancement,  and 
training  in  special  areas.  Norms  that  are  associated  with  the  various 
job  levels  provide  a  basis  for  job  classification. 

Zn  addition  to  using  the  test  as  an  instrument  of  individual 
evaluation,  it  can  also  be  used  as  one  for  company  evaluation.  Thac  Is, 
the  general  level  of  a  company's  computer  personnel  can  be  compared  to 
that  of  organisations  within  the  same  area  of  computer  application.  The 
weaknesses  and  strengths  of  the  company  personnel  might  be  explored 
through  examination  of  the  scores  on  particular  sets  of  items.  Recommen¬ 
dations  for  training  would  be  a  natural  consequence  of  such  an  examinatio 
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Fig.  1.  Plan  for  selecting  applicants  for  progranming  and  systems  analysis  positions. 


Certification  of  Prograamlng  Peraonnel 

r 

*  There  is  a  need  for  setting  up  proficiency  standards  in  the  computer 

programming  field.  The  BPKT  could  be  useful  as  a  certification  instrument 
In  the  computer  industry,  just  as  proficiency  tests  are  used  in  pro¬ 
fessional  fields  such  as  law  and  medicine. 

Evaluation  of  Training  Program  Ef fectlveneac 

There  is  frequently  a  large  disparity  between  an  organization's 
training  program  and  its  actual  job  requirements.  The  BPKT  could  be 
used  to  make  modifications  in  the  training  prograr  by  identifying  those 
areas  in  which  personnel  on-the-job  are  inadequately  prepared. 

Validation  of  Aptitude  Tests 

For  research  purposes,  the  BPKT  will  serve  as  a  criterion  for 
determining  how  well  aptitude  tests  can  predict  on-the-job  success. 

^  Thus  far,  training  grades  have  bean  the  main  basis  for  determining  the 

degree  of  validity  of  aptitude  teats.  Whan  ratings  and  other  such 
measures  have  bean  used  as  criteria,  the  predictiveness  of  aptitude 
tests  has  been  generally  low.  2t  is  uncertain  whether  these  results 
are  due  to  the  low  reliability  and  inapproprlatenaes  of  the  criterion 
measures  employed,  or  to  the  real  lack  of  validity  of  the  aptitude  tests. 
The  BPKT  will  serve  as  a  reliable  criterion  for  establishing  such 
validity. 
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SECTION  V.  FUTURE  PLANS 


Maintaining.  Updating,  and  Revising  the  BPKT 

The  BPKT  as  any  other  testing  instrument  must  be  maintained  and 
updated  for  its  most  effective  use.  Maintenance  of  the  present  form 
requires  that  the  base  for  NAVCOSSACT  and  other  group  norms  be  expanded 
for  better  interpretive  use  of  the  scores.  When  this  data  base  is 
expanded,  other  information  also  should  be  collected,  so  that  the  BPKT 
can  be  correlated  with  other  indices  and  predictors. 

The  updating  of  the  BPKT  is  necessary  to  keep  pace  with  the  changing 
technology  and  skill  requirements  of  the  computer  field.  New  test 
questions  should  be  tried  out  during  regular  test  administration  and 
should  replace  those  questions  whose  relevance  has  diminished. 

A  modification  of  the  BPKT  has  b««n  found  necessary  to  make  it 
more  suitable  for  testing  business-applications  programmers.  Thirty- 
five  test  questions  in  the  BPKT  will  be  replaced  to  make  the  test  more 
appropriate  for  this  area.  The  modified  form  should  be  ready  for  normlng 
within  the  next  year.  The  same  maintenance  and  updating  requirements 
would  apply  to  this  form,  as  wall. 

Special -Ares  Taft* 

The  BPKT  was  designed  to  discriminate  best  at  lower  and  Intermediate 
levels  of  programming  experience.  It  was  recognized  that  no  single 
criterion  measure  can  be  expected  to  discriminate  equally  well  across 
the  entire  range  of  competence  involved  in  programming  and  systems 
analysis.  A  second  type  of  criterion  measure  was  planned  to  provide 


better  discrimination  at  the  higher  job  levels.  This  second  type  of 
criterion  measure,  covering  the  areas  of  systems  analysis,  systems 
design,  and  program  design,  is  currently  under  development.  Each  test 
will  probe  in  depth  an  Individual's  proficiency  in  a  particular  area  of 
computer  programming  or  analysis.  These  areas  were  defined  by  the  job 
dimensions  found  in  the  job  analysis.  Consistent  with  the  skill  require¬ 
ments  of  each  job  dimension,  several  typical  problems  are  being  constructed 
for  each  test.  The  special-area  tests  will  require  the  individual  to 
work  out  solutions  to  fewer  but  more  complex  problems  than  those  included 
in  the  BPKT . 

A  sequence  of  six  steps  has  been  planned  to  complete  the  special- 
area  tests:  problem  construction,  problem  review  by  a  NAVCOSSACT 
committee,  pretests  of  test  problems,  a  final  review  by  a  NAVCOSSACT 
committee,  collection  of  normative  data  on  the  sample  previously  tested 
with  the  BPKT,  and  analysis  of  the  test  data.  A  technical  report  will 
present  results  and  recommendations  for  the  uee  of  this  type  of  criterion 
measure  in  conjunction  with  the  BPKT. 

For  reaearch  purposes,  the  systems  analysis  and  systems  design  tests 
will  serve  as  higher  level  criteria  for  validating  aptitude  tests  and 
selection  procedures.  These  special-area  tests  also  will  be  used,  in 
conjunction  with  the  BPKT,  in  personnel  selection  and  classification, 
personnel  evaluation  and  upgrading,  and  training  program  evaluation. 
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APPENDIX  A 

PROGRAMMER  TASKS 


Logic .  Estimation,  and  Analysis 


* 

i 


Task  Statement 

Number 

47  Express  solution  to  problem  as  sequence  of  logical  steps. 

48  Develop  logical  descriptions  of  functions  to  be  programmed . 

51  Specify  the  method  to  be  used,  from  among  the  available 

methods,  to  perform  a  calculation  or  function. 

55  Estimate  running  time  and  size  of  the  computer  program  to 

be  produced. 

57  Estimate  amount  of  computer  t'.ae  required  for  program 

check-out  purposes. 

76  Analyse  anticipated  program  flow  so  as  to  utilize  appro¬ 
priate  library  subroutines  as  opposed  to  writing  new 
subroutines. 

77  Analyse  frequency  of  use  of  subroutines  to  determine 
whether  a  "closed  subroutine"  or  an  "open  subroutine" 
should  be  used  and  program  accordingly. 

78  From  the  routines  available  for  a  given  method,  determine 
which  routines  fit  within  the  specific  requirements  of 
the  problem,  and  pick  one  of  these  routines. 

127  Study  and  analyse  program  description,  flow  charts,  and 

listings  as  a  prelude  to  program  modifications  or  revisions. 

130  Study  and  analyse  program  documentation  as  a  prelude  to 
the  integration  or  utilisation  of  the  program  in  a  progran 
system. 

131  Study  and  analyze  programing  language  manuals  (describing 
new  or  unfamiliar  programming  languages)  in  order  to  be¬ 
come  proficient  in  the  use  of  the  language. 
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Coding  Operations 


Task  Statement 
Number 

31  Translate  detailed  flow  diagrams  into  machine  language 
or  symbolic  language  instructions. 

32  Trans L  3  flow  diagrams  directly  into  machine  language 
or  symb^.ic  language  instructions  without  going  through 
stage  of  detailed  flow  diagrams. 

36  Code  from  program  specifications  in  an  intermediate 
symbolic  language. 

37  Code  in  symbolic  assembly  language  for  a  simple  one-to-one 
translation  into  machine  code. 

38  Code  in  a  symbolic  assembly  language  utilizing  pseudo¬ 
instructions,  macro-instructions,  and  systems  routines. 

39  Code  in  compiler  language. 


Documentation 

Task  Statement 
Number 


153 

Document 

program  requirements. 

155 

Document 

program  design  spec if lea t ions. 

157 

Document 

coding  speclflcetlone. 

160 

Document 

finalised  flow  chart. 

161 

Document 

program  description. 

164 

Document 

operating  instructions. 

165 

Document  coding  by  including  appropriate  comments  next 
to  source  language  instructions. 

166 

Write  program  changes  or  change  request  document,  e.g. 
error  reports,  revision},  modifications. 

Task  Statement 
Number 


Programing  Coustraints 


79  Estimate  minimum  and  maximum  values  In  computations  in 
order  to  avoid  overflow  stops,  loss  of  significant  digits, 
etc. 

80  Inspect  constants  and  computational  sequences  for  possible 
overflow  conditions. 

81  Decide  among  alternative  computational  procedures  for 
handling  specific  problems,  e.g.,  multiple  precision  vs. 
single  precision;  round-off  (or  rounding)  vs.  truncation* 
floating  vs.  fixed,  etc. 

84  Analyze  and  plan  piograms  so  as  t.o  minimize  required 

storage  space  and/or  time  (running  tine). 

87  Decide  among  alternative  formats  for  ’.nput  or  output  in 
formation. 

88  Design  the  program  to  meet  the  requirements  of  specified 
formats  of  input  and  output  information. 

91  Design  and  write  program  to  insure  that  program  size 
(number  of  registers)  does  not  exceed  space  limit. 

92  Allocate  memory  for  a  program  within  constraints  of 

llmltad  apaca  and  time. 

94  List  constants  »nd  parameters  and  specify  their  memory 

locations  if  necessary. 

93  Break  up  large  programs  into  self ’contained  pieces  thc>. 
can  be  accommodated  by  available  memory. 
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Task  Statement 
Number 


ProRrar.  Testing  and  Checking 


5  Devise  program  test  plan. 

6  Build  consistency  checks  Into  a  program.  Examples: 

(1)  to  check  whether  input  values  of  data  co  be 
processed  are  within  the  ranges  expected  by 
the  programmer. 

(2)  to  check  on  the  computed  value  of  a  payroll 
check  to  see  whether  it  is  less  than  zero. 

7  Construct  appropriate  test  data  fully  to  check  all  paths 
of  a  program. 

8  Locate  "breakpoints"  at  most  useful  locations  to  obtain 
debugging  information  or  data. 

9  Code,  in  any  language,  special-purpose  or  "one-shot" 
testing  routines  for  own  program, 

10  Debug  own  programs  written  in  machine  or  assembly  language, 
by  "desk-checking." 

11  Debug  at  machine  language  level  own  program  written  in 
automatic  coding  language. 

12  Debug  at  source  level  own  programs  written  in  automatic 
coding  language. 

13  Debug  at  machine  language  level  programs  of  other's 
written  in  an  automatic  coding  language. 

14  Debug  at  source  language  level  programs  written  by  otners 
in  automatic  coding  language. 

15  Debug,  by  "deck-checking",  programs  of  others  written  in 
machine  or  assembly  language. 

16  Spot  check  accuracy  of  own  or  other's  programs  away  from 
the  computer  by  checking  code  Itself,  using  such  cues  as 
logical  Inconsistencies  In  the  Instructions,  obviously 
Incorrect  addresses,  etc. 

18  Debug  owu  programs  by  "stepping  through"  at  console  Itself. 

19  Spot  check  own  or  other's  programs  for  accuracy,  regardless 
of  language  level,  by  checking  captured  Intermediate  results 
for  expected  ,i‘uec. 

40  Make  corrections  to  own  or  other's  programs  by  reassembly 
or  recompi’atlon  of  corrected  source  language  code. 

41  Make  corrections  to  own  or  other's  programs  by  numeric 
corrections  of  object  program. 

129  Study  and  analyze  program  documentation  as  a  prelude  to 

debugging  or  checkout  of  a  program  produced  by  someone 
else. 
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Flow  Diagramming 


Task  Statement 
Number 

25  Given  a  functional  description  of  program  requirements 
generate  functional  block  diagrams. 

26  Translate  functional  block  diagrams  directly  into  machine 
instructions  without  going  through  flow  and  detailed  flow 
diagramming  stages. 

>1  Given  program  design  specifications,  generate  program 

diagrams  or  flow  charts. 

28  Translate  broad  system  flow  charts  into  flow  diagrams 

with  elements  such  as: 


Compute 
A  =  B/D  +  C/E 


29  translate  functional  block  diagrams  into  fl  w  diagrams, 

with  elements  such  as: 


Compute 
A  *  B/D  +  C/E 


30  Translate  flow  diagrams  with  elements  such  j s; 


Compute 
A  «  B/D  +  C/E 


into  detailed  flow  diagram  eien-ents,  given  in  port  below: 


3i  Generate  a  flow  dia;  am  from  machine  language  or  symbolic 

language  instructions. 
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APPENDIX  B 

EDUCATIONAL  AND  VOCATIONAL  DATA  FORM 

1,  Tesc  Booklet  Number  _  Sex:  M  _  F  _  Age  _ 

C.5.  Level  or  Organization  and 

2.  Military  Rank  _ Division  _ 

3.  Highest  School  Grade  (circle.) 

High  Sw.^ol:  9  10  11  12  College:  1234  Post  Grad;  1234 

4,  Degrees 

Date  Major  Minor 


None  (year  left  college) 
Bachelor ' s 


Ma  s  t  e  r  '  s 


Doc toral 


3  Computer-related  Training  (check  one  or  both  columns  that  apply  to 
your  training. 


On-the-job 

Formal 

training 

course 

A. 

General  progr rntiiug  procedures 

(  ) 

( 

) 

K. 

System  analysis 

(  ) 

< 

) 

C. 

Numerical  analysis 

(  ) 

( 

) 

D. 

Computer  operation 

(  ) 

( 

) 

E. 

Machine  languages 

(  ) 

( 

) 

F . 

Mach  ins- independent 

languages 

(  ) 

( 

) 

G. 

Other  (Specify) 

(  ) 

( 

) 

No. 

of  years  since  laDt 

formal  classroom  course 

1.  Current  job  title  (give  full  citle(s)) 


3.  No.  of  months  with  present  job  title  (In  this  organization) 
Type  of  computer(s)  involved  in  p. esent  position: 


Programming  languages  you  use: 
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9.  List  previous  programning/analytical  job  titles  (in  this  and  other 
organizations).  Indicate  also  the  approximate  number  of  months  in 
each  and  the  types  of  computers  worked  with.  Start  with  most  recent 
and  work  back  to  first  job  in  computer  field. 

Previous  programing/ ana  lytic  No.  of  Types  of  computer (s) 
titles  Months  involved 


10,  Total  no.  of  months  in  computer-related  jobs  (#8  r.nd  # 9  combined); 


11. 


Which  of  the  following  are  major  parts  or  duties  of  your  current 
job?  Check  column  1  for  each  item  that  applies.  Of  those  checked 
in  column  1,  check  column  2  for  the  ONE  item  that  describes  the 
largest  part  of  your  job. 


1  2 


irogram  System  Analysis  (  ) 

Program  System  Design  (  ) 

Program  System  Integration  (  ) 

(Integrating  system  components)  (  ) 

Planning  and  Scheduling  Program  Production  (  ) 

Supervising  Program  Production  (  ) 

Lead  Programming  Responsibility  (  ) 

Detailed  Program  Design  and  Coding  (  ) 

Debugging  (Desk-checking)  (  ) 

Program  Testing  (Checking-out  individual  (  ) 

programs  or  sub- systems)  (  ) 

Program  System  Testing  (  ) 

Program  Documentation  (  ) 

Program  Installation  (Turnover)  (  ) 

Training  (of  other  personnel)  (  ) 

Other  (specify)  (  ) 


(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  ) 
(  > 
i  ) 


12.  A.  Which  of  the  following  programing  areas  have  you  been  involved 

with  to  a  significant  degree  in  your  current  job?  Check  column  1 
tor  each  area  that  applies.  Of  those  checked  in  column  1,  check 
column  2  for  the  ONE  that  comprises  the  major  area  of  your  work 

1  2 

Utility  Program  Development  (  )  (  ) 

(General  Purpose  end  Librety) 

Utility  Program  Development  (  )  (  ) 

(Executive  and  Compiler) 

Operational/ functional  Program  Development  (  )  (  ) 

(Math  or  Scientific) 

Operational/ functional  Program  Development  (  )  (  ) 

(Data  Manipulation,  e.g.,  inventory  control.  Information  Retrieve 

B.  Check  below  if  you  have  been  iuvol/ed  in  the  types  of  programming 
indicated  in  your  current  job. 


Programming  Real-time  Systems  (  ) 

Programming  lor  Special  Purpose  Computers  (  ) 
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